Augmented reality (AR)-capable headset and visually detected controller devices

ABSTRACT

Aspects described herein include a wearable headset comprising an optical arrangement configured to couple light from a display onto a first field of view, and one or more visual sensors defining a second field of view. The wearable headset further comprises one or more computer processors configured to receive a signal indicating that the headset is in an unworn state, and, while the headset is in the unworn state, operate the one or more visual sensors to detect one or more user inputs within the second field of view.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 16/659,222, filed Oct. 21, 2019 and is herein incorporated byreference in its entirety.

BACKGROUND

The present disclosure generally relates to computer-basedentertainment, and more specifically to augmented reality (AR)-capabledevices and visually detected controller devices.

Computer graphics technology has significantly progressed since thefirst video games were developed. Relatively inexpensive 3D graphicsengines now provide nearly photo-realistic interactive game play onhand-held video game, home video game, and personal computer hardwareplatforms costing only a few hundred dollars. These video game systemstypically include a hand-held controller, game controller, or, in thecase of a hand-held video game platform, an integrated controller. Auser interacts with the controller to send commands or otherinstructions to the video game system to control a video game or othersimulation. For example, the controller may include a joystick andbuttons operated by the user.

While video games allow the user to interact directly with the videogame system, such interactions primarily influence the graphicaldepiction shown on the video game device (or on a connected display),and rarely influence any other objects outside of the virtual world.That is, a user may specify an input to the video game system,indicating that the user's avatar should perform a jump action, and inresponse the video game system could display the user's avatar jumping.However, such interactions are typically limited to the virtual world,and any interactions outside the virtual world are limited (e.g., ahand-held gaming device could vibrate when certain actions occur).

Additionally, many hand-held gaming devices include some form of visualsensing device which may be used to capture an image or a series ofimages of a physical, real-world scene. The captured images can then bedisplayed, for instance, on a display of the hand-held gaming device.Certain devices may be configured to insert virtual objects into thecaptured images before the images are displayed. Additionally, otherdevices or applications may enable users to draw or paint particularwithin a captured image of a physical scene. However, as suchalterations apply only to a single image of the physical scene,subsequent captured images of the physical scene from differentperspectives may not incorporate the user's alterations.

SUMMARY

In one embodiment, a wearable headset comprises an optical arrangementconfigured to couple light from a display onto a first field of view,and one or more visual sensors defining a second field of view. Thewearable headset further comprises one or more computer processorsconfigured to receive a signal indicating that the headset is in anunworn state, and, while the headset is in the unworn state, operate theone or more visual sensors to detect one or more user inputs within thesecond field of view.

In another embodiment, a system comprises a docking device and awearable headset. The wearable headset comprises an optical arrangementconfigured to couple light from a display onto a first field of view,one or more visual sensors defining a second field of view, and one ormore computer processors. The one or more computer processors areconfigured to receive a signal indicating that the headset is dockedwith the docking device, and while the headset is docked with thedocking device, operate the one or more visual sensors to detect one ormore user inputs within the second field of view.

In another embodiment, a method is described for use with a wearableheadset. The method comprises receiving a signal indicating whether theheadset is an unworn state. The method further comprises, while theheadset is in a worn state, outputting visual content to a display ofthe headset, wherein the headset further comprises an opticalarrangement configured to couple light from the display onto a firstfield of view. The method further comprises, while the headset is in theunworn state, operating one or more visual sensors of the wearableheadset to detect one or more user inputs within a second field of viewof the one or more visual sensors.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited aspects are attained andcan be understood in detail, a more particular description ofembodiments of the disclosure, briefly summarized above, may be had byreference to the appended drawings. It is to be noted, however, that theappended drawings illustrate only typical embodiments of the disclosureand are therefore not to be considered limiting of its scope, for thedisclosure may admit to other equally effective embodiments.

FIG. 1 illustrates exemplary operation of an AR-capable headset andvisually detected controller devices, according to one or moreembodiments.

FIG. 2 is a diagram illustrating attachment of a mobile computing devicewith a mount assembly, according to one or more embodiments.

FIG. 3 illustrate an exemplary optical arrangement within a compactAR/VR display device, according to one or more embodiments.

FIG. 4 is a block diagram of an exemplary system, according to one ormore embodiments.

FIG. 5 is an exemplary method of operating an AR-capable headset,according to one or more embodiments.

FIG. 6A illustrates an exemplary docked state of an AR-capable headset,according to one or more embodiments.

FIG. 6B illustrates an exemplary docked state of an AR-capable headset,according to one or more embodiments.

FIG. 6C illustrates an exemplary docked state of an AR-capable headset,according to one or more embodiments.

FIG. 7 illustrates an exemplary AR mode, according to one or moreembodiments.

FIG. 8 illustrates an exemplary AR mode having a virtual two-dimensionaldisplay, according to one or more embodiments.

FIG. 9 illustrates an exemplary mode in which an unworn AR headsetvisually detects controller devices, according to one or moreembodiments.

DETAILED DESCRIPTION

An AR-capable headset may support a mixed reality experience, in whichthe headset displays visual content (e.g., graphics for a gameapplication) overlaying a view of the physical environment. One or morevisual sensors of the headset may visually track one or more controllerdevices that are hand-held or body-worn by a user. Input provided viathe controller device(s) (e.g., tracked movement of the controllerdevice(s), pressing button(s)) may be used to adapt the visual content.The headset often has minimal functionality when in an unworn state,e.g., the display is disabled while a battery of the headset isrecharging. Further, the headset may not support integration with anexternal display (that is, external to the headset), such as atelevision (TV).

Embodiments described herein include an AR-capable headset comprisingone or more visual sensors and one or more computer processors. The oneor more computer processors are configured to determine whether theheadset is in an unworn state, wherein the headset has a substantiallystatic orientation. While the headset is in the unworn state, the one ormore computer processors are further configured to receive input fromone or more controller devices, output visual content to an externaldisplay, and visually detect the one or more controller devices usingthe one or more visual sensors. In some embodiments, a docking devicearranges the headset with the substantially static orientation, anddetermining whether the headset is in an unworn state comprisesdetermining whether the headset is docked with the docking device.

Thus, the headset supports multiple modes of user interaction, which mayinclude displaying AR visual content in one or more modes while a useris wearing the headset, and displaying visual content in one or moreother modes while a user is not wearing the headset. For example, theheadset may display three-dimensional AR visual content in a first modewhile the user is wearing the headset, and may in a second mode displayvisual content using an external display after the user removes theheadset. While unworn, the headset may be capable of receiving input toadapt the visual content, e.g., one or more visual sensors of theheadset may visually track the controller devices.

The headset may further support software applications that seamlesslytransition visual presentation between the display included in theheadset and one or more external displays. In one example, a game mayrequire a user to wear the headset to view AR visual content related toa particular objective, then require the user to remove the headset forcontinued gameplay with the controller devices related to anotherobjective. In another example, an application may display a movie to anexternal display while the headset is unworn, then continue the movie onan AR virtual display when the user dons the headset. Other applicationsthat exploit the features of the headset will be apparent to one ofordinary skill in the art.

Although the headset is discussed primarily in terms of an ARcapability, it will be noted that the AR-capable headset may havefurther capabilities. In some embodiments, the AR-capable headset may beoperable in a virtual reality (VR) mode, which generally isolates a userwearing the AR-capable headset from their environment by replacing thenatural field of view of the environment with virtual imagery.

FIG. 1 illustrates exemplary operation of an AR-capable headset 110 andvisually detected controller devices 115-1, 115-2, according to one ormore embodiments. In diagram 100, a user 105 wears the headset 110 onhis/her head, and holds controller devices 115-1, 115-2 in his/herhands. The headset 110 comprises a lens 120 positioned in front of theeyes of the user 105, and a strap 125 attached to the lens 120 andconfigured to retain the headset 110 on the head of the user 105 in thedesired position.

A display (not shown) of the headset 110 displays visual content that isviewed by the user 105. One or more computer processors (not shown) ofthe headset 110 may execute one or more applications that provide thevisual content to be displayed by the display. One or more visualsensors (not shown) of the headset 110 acquire imagery of a portion ofthe physical environment surrounding the user 105. In some embodiments,the one or more visual sensors visually detect the controller devices115-1, 115-2. The one or more computer processors may receive user inputthat may be used to interactively adapt the visual content. The userinput may be received from the controller devices 115-1, 115-2, e.g., bythe user moving the controller devices 115-1, 115-2 (whether visuallydetected by the one or more visual sensors, responsive to signals fromsensors of the controller devices 115-1, 115-2, etc.), pressing buttonsof the controller devices 115-1, 115-2, and so forth.

In some embodiments, the display, the one or more computer processors,and the one or more visual sensors are integrated into, and notremovable from, the headset 110. However, in other embodiments, one ormore of the display, the one or more computer processors, and the one ormore visual sensors are removably attached to the headset 110.

In the example shown in FIG. 2, a mobile computing device 205 isremovably attached with a mount assembly 210, according to one or moreembodiments. The features illustrated in the diagram 200 may be used inconjunction with other embodiments, e.g., the mount assembly may beintegrated into the headset 110 of FIG. 1. The mobile computing device205 may be implemented in any suitable form, such as a smartphone or atablet computer. The mobile computing device 205 generally comprises oneor more computer processors and a memory (not shown), and a display 215using any suitable display technology.

The diagram 200 depicts an exemplary sequence for inserting the mobilecomputing device 205 into the mount assembly 210. The mount assembly 210may be formed of one or more elements of any material having suitablestrength for retaining the mobile computing device 205. In someembodiments, the mount assembly 210 is formed of a plastic material,which advantageously provides a lighter display device overall.

From a first position 225, the mobile computing device 205 is insertedthrough an opening 220 formed in the mount assembly 210. Theintermediate position 230 represents a possible positioning of themobile computing device 205 before reaching a predefined final position235. At the predefined final position 235 of the mobile computing device205, the display 215 of the mobile computing device 205 has a predefinedposition 240 relative to an optical arrangement 255.

A lower surface 245 of the mount assembly 210 is generally opticallytransmissive of light 250 generated by the display 215. In someembodiments, the lower surface 245 is formed of an opticallytransmissive material, such as a plastic or glass, through which thelight 250 from the display 215 is transmitted. In other embodiments, thelower surface 245 defines an opening through which the light 250 fromthe display 215 is transmitted. For example, the lower surface 245 maysupport the mobile computing device 205 around a periphery of the mobilecomputing device 205.

Although not explicitly shown, the mount assembly 210 may includefurther elements for removably attaching the mobile computing device 205with the mount assembly 210. For example, a press fit may be formedbetween the mobile computing device 205 and the mount assembly 210 usingadjustable corner piece(s), a sliding tray with guide plug, togglepin(s), a stepped slot, a replaceable tray, etc. For example, the mobilecomputing device 205 may be inserted into a replaceable tray or othersuitable carrier member, which is then inserted to the mount assembly210 to thereby arrange the display 215 with the predefined position 240.In this way, different carrier members may be used to accommodatedifferent types of mobile computing devices 205 for a particular mountassembly 210.

The removable attachment of the mobile computing device 205 with themount assembly 210 may have any suitable orientation within anassociated display device. The elements of the optical arrangement 255collectively define a field of view relative to predefined opticalreference point(s), and the display device is generally designed suchthat the eye(s) of a user are aligned with the optical referencepoint(s). To support an AR capability of the display device, the mobilecomputing device 205 and mount assembly 210 are generally disposedoutside of the field of view to allow a user to observe the physicalenvironment through the optical arrangement 255. For example, for ahead-worn display device in which a line of sight of the usercorresponds to the field of view of the optical arrangement 255, themobile computing device 205 and the mount assembly 210 may be positionedabove, below, or to a side of the user's line of sight.

Although the display 215 is described in terms of being implemented in amobile computing device 205, other implementations of the display 215are also contemplated. In other embodiments, the display 215 andcomputing device may both be fully integrated into and not removablefrom the headset 110. In other embodiments, the mobile computing device205 is separate from the display 215 that is fully integrated into theheadset 110.

FIG. 3 illustrates an exemplary implementation of a compact AR/VRdisplay device 300, according to one embodiment. The featuresillustrated in FIG. 3 may be used in conjunction with other embodiments,e.g., integrated into the headset 110 of FIG. 1. More specifically, thedisplay device 300 depicts an exemplary arrangement of the mobilecomputing device 205 and an optical arrangement 305 (one example of theoptical arrangement 255 of FIG. 2) after placement of the mobilecomputing device 205 in the mount assembly 210 (e.g., as depicted inFIG. 2).

The optical arrangement 305 is configured to reflect at least a portionof light 320 from the display 215 to an eye 355 of a user. The elementsof the optical arrangement 305 collectively define a field of view 360relative to a predefined optical reference point. The display device 300is generally designed such that the eye 355 of the user is aligned withthe optical reference point, e.g., when worn by the user.

As shown, the mobile computing device 205 is arranged with the display215 facing in a downward direction (that is, relative to the perspectiveof the user). However, alternate orientations of the display 215relative to the user's perspective are also possible, such asupward-facing, sideways-facing, etc. The optical arrangement 305 definesa first optical path 365-1 that extends from the display 215 to a beamsplitter element 310, from the beam splitter element 310 to a surface ofa mirror element 315, and from the mirror element 315 through the beamsplitter element 310 to the optical reference point. The opticalarrangement 305 further defines a second optical path 365-2 extendingfrom one or more objects in the environment through the mirror element315 and through the beam splitter element 310 to the optical referencepoint.

As shown, the light 320 (or “imagery”) that is generated by the display215 is transmitted in the downward direction towards the beam splitterelement 310. In some embodiments, the display 215 is a non-polarizeddisplay and the light 320 is non-polarized. In other embodiments, thedisplay 215 is a polarized display and the light 320 is polarized.

Based on the light 320, a first incident light 325 is incident on thebeam splitter element 310, and a first portion 330 of the first incidentlight 325 is reflected onto the field of view 360. In some embodiments,the beam splitter element 310 reflects approximately 50% of the firstincident light 325 as the first portion 330 (e.g., where the beamsplitter element 310 is implemented as a half-silvered mirror), andtransmits approximately 50% of the first incident light 325 through thebeam splitter element 310, although other ratios of reflection andtransmission are also possible.

A second incident light 335 is incident on the mirror element 315, and asecond portion 340 is reflected by the mirror element 315 toward thebeam splitter element 310. The mirror element 315 may have any suitablereflection/transmission ratios for reflecting the second incident light335 and transmitting light from the environment. In one embodiment, themirror element 315 comprises a film (e.g., a coating or other reflectivematerial applied to an inner curvature of the mirror element 315) thatdefines a partially reflective surface facing the beam splitter element310. In some embodiments, the mirror element 315 reflects between about50% and 80% of the second incident light 335 as the second portion 340,although other ratios of reflection and transmission are also possible.In one embodiment, the mirror element 315 reflects about 67% of thesecond incident light 335. A third incident light 345 is incident on thebeam splitter element 310, and a third portion 350 is transmittedthrough the beam splitter element 310 to the optical reference point.

As mentioned above, in some embodiments the display 215 may be apolarized display and the light 320 is polarized. In some embodiments,the light 320 is linearly polarized (whether in s-polarization orp-polarization), and the beam splitter element 310 has a polarizationaxis that is rotated a net 90° from the polarization axis of the light320. Here, because the light 320 has a polarization axis that is rotatedfrom the polarization axis of the beam splitter element 310, the firstportion 330 reflected by the polarized beam splitter element 310comprises a majority (e.g., nearly all) of the first incident light 325.

Before reaching the mirror element 315, the first portion 330 passesthrough a quarter-wave plate element 370 (or “quarter-wave retarder”),which transforms the linear polarization of the first portion 330 into acircular polarization. The circularly polarized light is incident on themirror element 315 as the second incident light 335, and the secondportion 340 is reflected off the mirror element 315. The second portion340 passes through the quarter-wave plate element 370, which transformsthe circularly polarized light into linearly polarized light with a net90°-rotated polarization from the polarization axis of the polarizedbeam splitter element 310. The third incident light 345 is incident onthe beam splitter element 310, and the third portion 350 is transmittedby the beam splitter element 310. Because the polarization axis of thethird incident light 345 and the polarization axis of the beam splitterelement 310 are aligned, the third portion 350 comprises a majority ofthe third incident light 345. In this way, losses may be reduced at eachincidence of the light on the beam splitter element 310.

Such an implementation may also increase the amount of light frombackground objects that is transmitted through the mirror element 315and the beam splitter element 310 to the user's eye 355. When light frombackground objects passes through a polarized surface of the mirrorelement 315 (e.g., a chemical film applied to the mirror element 315),the light becomes linearly polarized and the quarter-wave plate element370 transforms the polarization of this light so that a largerpercentage passes through the beam splitter element 310. The net effectof adding the polarizing reflective surface to the mirror element 315,in combination with the quarter-wave plate element 370, is tosignificantly increase the amount of light that reaches the user's eye355, including the light 320 emitted from the display 215 and light fromthe background objects viewed through the optics of the opticalarrangement 305.

Although not shown, the optical arrangement 305 may further include amask that is configured to block light from some of the display area ofthe display 215 and/or from other portions of the mobile computingdevice 205 to prevent these portions from being seen by the user. Forexample, a mask may be provided to prevent edges of the mobile computingdevice 205 from being visible within the field of view 360, which tendsto distract the user from the immersive nature of the interactiveenvironment.

Although not shown, the optical arrangement 305 may further include alight-blocking assembly disposed within the field of view 360. In someembodiments, the light-blocking assembly comprises cross polarizers.When one or both of the cross polarizers are rotated, the amount oflight from the physical environment that is transmitted to the user'seyes (e.g., through the beam splitter element 310) can be controlled tosubstantially isolate the field of view 360 from the physicalenvironment (e.g., corresponding to a selected VR mode). Stated anotherway, the light-blocking assembly may be used to selectively isolate thesecond optical path 365-2. Rotating the cross polarizers may beperformed manually (e.g., the user turns a knob linked with the crosspolarizers) or electronically. For example, a motor linked with thecross polarizers receives control signals from an associated computingdevice (such as the mobile computing device 205) and rotates the crosspolarizers based on a selected AR or VR display mode. In otherembodiments, the light-blocking assembly includes a partially or fullytransmissive “see-through” display, such as an OLED or a side-lit ornaturally-lit LCD. In this case, the partially or fully transmissivedisplay receives control signals from the associated computing deviceand selectively darkens the display based on the selected AR or VRdisplay mode.

Although not shown, a camera included in the mobile computing device 205may be used as a visual sensor of the headset 110 of FIG. 1. Forexample, the camera may be included on an opposite surface from thedisplay 215. In some embodiments, the display device 300 may furtherinclude a second mirror element configured to reorient a sensing axis ofthe camera. In some embodiments, the camera senses in the forwarddirection corresponding to an axis of the field of view 360. In thisorientation, the camera is able to acquire visual information for theenvironment for performing optical detection and tracking, depthestimation, and so forth. The second mirror element can be as simple asa single 90° fold mirror, or can be more complex including multiplemirrors and/or different mirror curvatures. In another implementation,the camera of the mobile computing device 205 may be included on thesame surface as the display 215.

FIG. 4 is a block diagram of an exemplary system 400, according to oneor more embodiments. The features illustrated in the system 400 may beused in conjunction with other embodiments.

The headset 110 comprises one or more computer processors 405 and amemory 410. The one or more computer processors 405 may be implementedin any suitable form, such as a general purpose microprocessor, acontroller, an application-specific integrated circuit (ASIC), and soforth. The memory 410 may include a variety of computer-readable mediaselected for their size, relative performance, or other capabilities:volatile and/or non-volatile media, removable and/or non-removablemedia, etc.

The memory 410 may include one or more applications (or modules) forperforming various functions described herein. In one embodiment, eachapplication includes program code that is executable by the one or morecomputer processors 405. However, other embodiments may include modulesthat are partially or fully implemented in hardware (i.e., circuitry) orfirmware of the headset 110. As shown, the memory 410 comprises anapplication 412 that generates visual content to be displayed based onthe state of the headset 110 (e.g., worn or unworn, docked, etc.), inputreceived from controller devices 115-1, 115-2, . . . , 115-N, and soforth. For example, the application 412 may be implemented as a gameproviding a dynamic gameplay environment for one or more users, and mayhave any suitable form such as an action game, an adventure game, arole-playing game, a simulation game, a strategy game, a sports game, aparty game, a trivia game, an educational game, and so forth. In anotherexample, the application 412 may be implemented as a streaming mediaapplication.

The headset 110 further comprises one or more sensors 415communicatively coupled with the one or more computer processors 405.Sensor signals from the one or more sensors 415 may be used as input bythe one or more computer processors 405 to determine the state of theheadset 110, to interactively adapt the visual content, and so forth.The one or more sensors 415 may be implemented in any form. In someembodiments, the one or more sensors 415 comprise one or more visualsensors 417. The one or more visual sensors 417 may have any suitableimplementation, such as a visual camera or an infrared camera. Forexample, the one or more visual sensors 417 may comprise a camera havinga forward-oriented field of view when the headset 110 is in a wornstate. In some embodiments, the one or more sensors 415 further comprisea worn-state sensor indicating whether the headset 110 is being worn.Some examples of the worn-state sensor include a sensor that detectscontact with a user's face, a photodetector that receives light from anLED when the headset 110 is not being worn (e.g., the user's face is notblocking the path between the LED and photodetector), and so forth.Other types of sensors are also contemplated for the one or more sensors415, such as a navigation system (e.g., a Global Positioning Systemreceiver), an inertial measurement unit (IMU), and so forth.

The headset 110 further comprises a display 215 communicatively coupledwith the one or more computer processors 405. The display 215 mayinclude any type of dynamic display capable of displaying a visualinterface to a user of the headset 110. Some non-limiting examples ofsuitable display technologies include any type of light emitting diode(LED), organic LED (OLED), liquid crystal display (LCD), plasma, andelectroluminescence (EL). As discussed above, the display 215 may beintegrated into the headset 110, may be detachable from the headset 110,may be integrated into a mobile computing device, and so forth.

The headset 110 further comprises a power source 420 that provideselectrical power to components of the headset 110. The power source 420may be implemented in any suitable form, such as a rechargeable orreplaceable battery. In some embodiments, the power source 420 is arechargeable battery that may be recharged via a conductive connectionor via a wireless connection (e.g., inductive charging).

The system 400 further comprises one or more controller devices 115-1,115-2, . . . , 115-N (generically referred to as controller device(s)115). The controller devices 115 may have any suitable implementation.In some embodiments, the one or more controller devices 115 arehand-held or body-worn. For example, a controller device 115 may be wornas a glove on a user's hand, as a band or sleeve on a user's arm, and soforth. Other modes of holding or wearing the controller devices 115 arealso contemplated.

Each controller device 115 comprises one or more computer processors445, a memory 450, one or more sensors 455, and a power source 460. Theone or more computer processors 445 may be implemented in any suitableform. The memory 450 may include a variety of computer-readable mediaselected for their size, relative performance, or other capabilities.The one or more sensors 455 are communicatively coupled with the one ormore computer processors 445. In some embodiments, the one or moresensors 455 comprises an IMU 457, which may include one or more of anaccelerometer, a gyroscope, and a magnetometer that acquire orientationdata and/or movement data of the controller device 115. Other types ofsensors are also contemplated.

Each controller device 115 further comprises a power source 460 thatprovides electrical power to components of the controller device 115.The power source 460 may be implemented in any suitable form, such as arechargeable or replaceable battery.

In some embodiments, each controller device 115 comprises one or morevisual reference features 465 arranged on one or more external surfacesof the controller device 115. The one or more visual reference features465 may be disposed on the external surface, or may be disposed withinthe controller device 115 and visible at the external surface (e.g.,light that is transmitted through an optically permissive externalsurface, light that is transmitted through an opening formed in theexternal surface).

In some embodiments, the one or more visual reference features 465comprise one or more passive visual indicators that are not electricallypowered, and/or one or more active visual indicators that areelectrically powered. Some non-limiting examples of the one or morepassive visual indicators include fiducials or other distinct featuresor markings at the external surface. The one or more passive visualindicators may be formed in the external surface and/or attached thereto(e.g., a sticker applied to the external surface).

One non-limiting example of the one or more active visual indicatorsincludes light emitting diodes (LEDs) that emit light in the visiblespectrum or infrared spectrum. In some embodiments, the one or morecomputer processors 445 control the visual appearance (e.g.,arrangement, color) of the one or more active visual indicators tovisually distinguish the controller device 115. In some embodiments, thevisual appearance of the one or more active visual indicators iscontrolled based on detecting a presence of other controller devices115. For example, the color(s) of the one or more active visualindicators may be selected as different colors to visually distinguishthe different controller devices 115.

In some embodiments, each controller device 115 further comprises one ormore input devices 490 communicatively coupled with the one or morecomputer processors 445. The one or more input devices 490 receive userinput during operation of the controller device 115, which may becommunicated by the controller device 115 to interactively adapt visualcontent displayed by the headset 110. The one or more input devices 490may be implemented in any suitable form, such as buttons, switches,triggers, touch sensors, and so forth.

The one or more computer processors 445 of the controller device 115 arecommunicatively coupled with the one or more computer processors 405 ofthe headset 110 via a communicative connection 470. The communicativeconnection 470 may be implemented in any suitable form, such aswireline, wireless, optical, etc. In some embodiments, the communicativeconnection 470 may differ based on an operational state of the headset110. In one example, while the headset 110 is in a worn state or in anunworn state (e.g., a docked state), and the controller device 115 isheld by the user, the communicative connection 470 may be wireless. Inanother example, while the headset 110 is in an unworn state (e.g., in adocked state) and the controller device 115 is not held by the user(e.g., in a docked state), the communicative connection 470 may bewireless or wired.

The one or more visual sensors 417 of the headset 110 have acommunicative connection 485 with the controller devices 115. Statedanother way, the one or more visual sensors 417 visually detect thecontroller devices 115 when the headset 110 is in a worn state and whenthe headset 110 is in an unworn state. The one or more computerprocessors 405 may perform image processing to identify positioningand/or motion of the controller devices 115. In some embodiments, visualdetecting the controller devices 115 comprises detecting the one or morevisual reference features 465.

In some embodiments, when the headset 110 is in a docked state (e.g., anunworn state in which the headset 110 has a substantially staticorientation), the headset 110 visually detects the controller devices115 using the one or more visual sensors 417. Stated another way, whenthe headset 110 is in the substantially static orientation, the one ormore visual sensors 417 are arranged to visually detect the controllerdevices 115.

The headset 110 may physically couple with a docking device 425. In someembodiments, the docking device 425 arranges the headset 110 with thesubstantially static orientation. The docking device 425 may beimplemented in any suitable form. For example, the docking device 425may define an exterior surface on which the headset 110 may be removablyplaced. In some embodiments, the docking device 425 may include one ormore features that mate or otherwise engage with one or more features ofthe headset 110. In some embodiments, the docking device 425 may retainthe headset 110 to maintain the substantially static orientation.

The headset 110 may communicatively couple with the docking device 425.In some embodiments, the docking device 425 comprises one or morecomputer processors 430 and a memory 435, which may be implemented inany suitable form. For example, the one or more computer processors 430may comprise a graphics processing unit (GPU). In some embodiments, theone or more computer processors 405 of the headset 110 arecommunicatively coupled with the one or more computer processors 430 viaa communicative connection 475. The communicative connection 475 may beimplemented in any suitable form, such as wireline, wireless, optical,etc.

In some embodiments, the one or more computer processors 405 identifywhether the headset 110 is in a docked state or not using thecommunicative connection 475. In one example, the one or more computerprocessors 405 may identify the docked state by detecting a presence ofthe communicative connection 475 with the docking device 425. In anotherexample, the one or more computer processors 405 may identify the dockedstate by detecting identification information of the docking device 425(e.g., an identifier stored in the memory 435).

In some embodiments, the docking device 425 comprises one or moreconnectors that physically engage with one or more connectors of theheadset 110 when the headset 110 is in the docked state. In someembodiments, the docking device 425 comprises circuitry that wirelesslycouples with circuitry of the headset 110 when the headset 110 is in thedocked state.

In some embodiments, the docking device 425 includes externalconnectivity that is not included in the headset 110, such that theconnectivity of the headset 110 may be expanded when the one or morecomputer processors 405 are communicatively coupled with the one or morecomputer processors 430. For example, the docking device 425 may includeone or more connectors to connect to an external display, to externalinput devices, and so forth.

In some embodiments, the computing resources of the docking device 425may be utilized by the headset 110. In some embodiments, the application412 may offload some of its computing tasks to be performed by the oneor more computer processors 430. In other embodiments, computing tasksof the application 412 are processed in parallel by the one or morecomputer processors 405 and the one or more computer processors 430. Insome embodiments, additional features of the application 412 may beenabled using the computing resources of the docking device 425. Forexample, the one or more computer processors 430 may allow theapplication 412 to output visual content with enhanced graphics.

In some embodiments, the docking device 425 delivers electrical power tothe headset 110 when the headset 110 is in the docked state. Theelectrical power may be used to charge the power source 420 of theheadset 110. For example, the docking device 425 may comprise one ormore conductive connections and/or one or more wireless connections(e.g., an induction coil for inductive charging) for delivering theelectrical power. In other embodiments, the headset 110 in the dockedstate receives electrical power from another source, such as wall power.

In some embodiments, one or more controller devices 115 may dockdirectly and/or indirectly with the docking device 425. When in a dockedstate, the one or more controller devices 115 may receive electricalpower to charge the respective power source 460. In one example, the oneor more controller devices 115 may mate or otherwise engage directlywith the docking device 425. In another example, the one or morecontroller devices 115 may mate or otherwise engage with the headset 110when the headset 110 is in an unworn state, and may be indirectly dockedwith the docking device 425 when the headset 110 is docked with thedocking device 425.

The one or more computer processors 405 may be communicatively coupledwith an external display 440 via a communicative connection 480. Theexternal display 440 may have any suitable implementation using anysuitable display technology, such as LED, OLED, LCD, plasma, EL, and soforth. The communicative connection 480 may be implemented in anysuitable form, such as wireline, wireless, optical, etc. In someembodiments, the one or more computer processors 405 output visualcontent to the display 215 when the headset 110 is in a worn state, andoutput visual content to the external display 440 when the headset 110is in an unworn state (e.g., a docked state). The one or more computerprocessors 405 may be communicatively coupled with the external display440 directly or indirectly (e.g., via the docking device 425).

FIG. 5 is an exemplary method 500 of operating an AR-capable headset,according to one or more embodiments. The method 500 may be performed inconjunction with other embodiments, such as being performed using one ormore computer processors of the headset 110 illustrated in FIGS. 1 and4.

The method 500 begins at block 505, where the one or more computerprocessors determine whether the headset is in an unworn state. In someembodiments, determining whether the headset is in an unworn state isbased on one or more sensor signals.

At block 515, when the headset is in an unworn state (“YES”), the method500 proceeds to block 520, where the one or more computer processorsreceive input from one or more controller devices. At block 530, the oneor more computer processors output visual content to an externaldisplay. At block 540, the one or more computer processors visuallydetect the one or more controller devices using the one or more visualsensors.

At block 515, when the headset is not in an unworn state (“NO”), themethod 500 proceeds to block 525, where the one or more computerprocessors receive input from the one or more controller devices. Atblock 535, the one or more computer processors output visual content toa display of the headset. At block 545, the one or more computerprocessors visually detect the one or more controller devices using theone or more visual sensors. The method 500 ends following completion ofone of block 540 or block 545.

FIGS. 6A-6C illustrate exemplary docked states of an AR-capable headset,according to one or more embodiments. The features illustrated in FIGS.6A-6C may be used in conjunction with other embodiments.

Referring to FIG. 6A, in diagram 600, the headset 110 and controllerdevices 115-1, 115-2 are in a first arrangement 620, in which thecontroller devices 115-1, 115-2 are removably attached to the headset110. As shown, the first arrangement 620 is disposed on a console table610 on a floor 615, and is disposed near a television 605 (one exampleof an external display 440).

In the first arrangement 620, the headset 110 is in a docked state 625.As shown, the headset 110 is oriented such that a camera, which has aforward-oriented field of view when the headset 110 is in a worn state,has a field of view that is generally oriented toward the user (e.g.,when viewing the television 605). In this way, the camera is capable ofvisually detecting the controller devices 115-1, 115-2 as the useroperates the controller devices 115-1, 115-2.

In some embodiments, the headset 110 may be connected to wall power whenin the docked state 625, and may provide electrical power to charge thecontroller devices 115-1, 115-2. Additionally, in the docked state 625,the headset 110 may be communicatively coupled with the television 605via the communicative connection 480, which may be wireline, wireless,etc.

Referring to FIG. 6B, in another arrangement 630, the headset 110 andthe controller devices 115-1, 115-2 are individually docked with thedocking device 425, which places the headset 110 in the docked state625. In some embodiments, the docking device 425 may be connected towall power, and may provide electrical power to charge the headset 110and/or the controller devices 115-1, 115-2. Additionally, the headset110 may be communicatively coupled with the television 605 directly viaa communicative connection 480-1, or through the docking device 425 viaa communicative connection 480-2.

Referring to FIG. 6C, in another arrangement 635, the controller devices115-1, 115-2 are removably attached with the headset 110. In turn, theheadset 110 is docked with the docking device 425, placing the headset110 in the docked state 625. In some embodiments, the docking device 425may be connected to wall power, and may provide electrical power tocharge the headset 110 and/or the controller devices 115-1, 115-2.Additionally, the headset 110 may be communicatively coupled with thetelevision 605 directly via the communicative connection 480-1, orthrough the docking device 425 via the communicative connection 480-2.

FIG. 7 illustrates an exemplary AR mode, according to one or moreembodiments. More specifically, diagram 700 illustrates an AR modedisplaying interactive virtual objects in three-dimensional space. Thefeatures illustrated in the diagram 700 may be used in conjunction withother embodiments, such as representing a view of a user when wearingthe headset 110 illustrated in FIGS. 1 and 4.

Diagram 700 includes a view 705 that represents the field of view of theuser when wearing an AR-capable headset. Within the view 705, visualcontent is generated by one or more computer processors to establish ascene. As shown, the view 705 includes a scene of virtual swordfighting,having a virtual opponent 710 and a virtual meter 715 indicating avirtual health of the virtual opponent 710.

The view 705 also includes portions of hands 720-1, 720-2 of the user,which are holding the controller devices 115-1, 115-2, respectively.Although not shown, the view 705 may further include other objects inthe physical environment. The controller device 115-1 includes a visualreference feature 465-1 as well as buttons 725-1. The controller device115-2 includes a visual reference feature 465-2 as well as buttons725-2. The buttons 725-1, 725-2 may be pressed by the user to provideinput to the one or more computer processors, e.g., to adapt thedisplayed visual content. The visual reference features 465-1, 465-2 arevisually detected by one or more visual sensors of the headset. Thepositioning and/or movement of the controller devices 115-1, 115-2 mayalso be used to provide input to the one or more computer processors.Virtual blades 730-1, 730-2 are displayed in the view 705 and arepositioned referenced to the visual reference features 465-1, 465-2. Thevirtual blades 730-1, 730-2 are positioned to appear to extend from thecontroller devices 115-1, 115-2, such that the controller devices 115-1,115-2 appear to be handles for the user's virtual swords.

FIG. 8 illustrates an exemplary AR mode having a virtual two-dimensionaldisplay, according to one or more embodiments. More specifically,diagram 800 illustrates an AR mode displaying virtual objects on thevirtual two-dimensional display. The features illustrated in the diagram800 may be used in conjunction with other embodiments, such asrepresenting a view of a user when wearing the headset 110 illustratedin FIGS. 1 and 4.

Within the view 705, a virtual character 810 and a virtual meter 815 aredisplayed on a virtual display 805. In some embodiments, the virtualdisplay 805 is a two-dimensional display, e.g., having the appearance ofa television or a computer monitor. In other embodiments, the virtualdisplay 805 may display content using multiple virtual surfaces. Thevirtual display 805 may be sized to provide a larger display than anexternal display (e.g., to provide a more immersive experience for theuser).

In the AR mode illustrated in the diagram 800, the buttons 725-1, 725-2may be pressed by the user to provide input to the one or more computerprocessors, e.g., to adapt the displayed visual content. The positioningand/or movement of the controller devices 115-1, 115-2 may also be usedto provide input to the one or more computer processors.

FIG. 9 illustrates an exemplary mode in which an unworn AR headsetvisually detects controller devices, according to one or moreembodiments. The features illustrated in diagram 900 may be used inconjunction with other embodiments.

In the diagram 900, the headset 110 is in an arrangement 905 disposed onthe console table 610 near the television 605. In the arrangement 905,the headset 110 is in a docked state 625. As shown, the headset 110 isoriented such that a camera, which has a forward-oriented field of viewwhen the headset 110 is in a worn state, has a field of view that isgenerally oriented toward the user (e.g., when viewing the television605). In this way, the camera is capable of visually detecting thecontroller devices 115-1, 115-2 as the user operates the controllerdevices 115-1, 115-2.

In some embodiments, the controller devices 115-1, 115-2 include visualreference features 465-3, 465-4, respectively, arranged on externalsurfaces of the controller devices 115-1, 115-2. As shown, the visualreference feature 465-1 is arranged on a first external surface of thecontroller device 115-1, and the visual reference feature 465-3 isarranged on a second external surface of the controller device 115-1opposite the first external surface. The visual reference feature 465-2is arranged on a third external surface of the controller device 115-2,and the visual reference feature 465-4 is arranged on a fourth externalsurface of the controller device 115-2 opposite the third externalsurface. Other arrangements of the visual references features 465-1, . .. , 465-4 are also contemplated.

In some embodiments, the headset 110 may be connected to wall power whenin the docked state 625. Additionally, in the docked state 625, theheadset 110 is communicatively coupled with the television 605. Thevisual content is provided from the headset 110 to the television 605.As shown, the visual content includes a virtual opponent 910 and avirtual meter 915.

In the mode illustrated in the diagram 900, the buttons 725-1, 725-2 maybe pressed by the user to provide input to the one or more computerprocessors, e.g., to adapt the displayed visual content. The positioningand/or movement of the controller devices 115-1, 115-2 may also be usedto provide input to the one or more computer processors.

In the current disclosure, reference is made to various embodiments.However, it should be understood that the present disclosure is notlimited to specific described embodiments. Instead, any combination ofthe preceding features and elements, whether related to differentembodiments or not, is contemplated to implement and practice theteachings provided herein. Additionally, when elements of theembodiments are described in the form of “at least one of A and B,” itwill be understood that embodiments including element A exclusively,including element B exclusively, and including element A and B are eachcontemplated. Furthermore, although some embodiments may achieveadvantages over other possible solutions or over the prior art, whetheror not a particular advantage is achieved by a given embodiment is notlimiting of the present disclosure. Thus, the aspects, features,embodiments and advantages disclosed herein are merely illustrative andare not considered elements or limitations of the appended claims exceptwhere explicitly recited in a claim(s). Likewise, reference to “theinvention” shall not be construed as a generalization of any inventivesubject matter disclosed herein and shall not be considered to be anelement or limitation of the appended claims except where explicitlyrecited in a claim(s).

As will be appreciated by one skilled in the art, embodiments describedherein may be embodied as a system, method, or computer program product.Accordingly, embodiments may take the form of an entirely hardwareembodiment, an entirely software embodiment (including firmware,resident software, micro-code, etc.) or an embodiment combining softwareand hardware aspects that may all generally be referred to herein as a“circuit,” “module” or “system.” Furthermore, embodiments describedherein may take the form of a computer program product embodied in oneor more computer readable medium(s) having computer readable programcode embodied thereon.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for embodiments of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present disclosure are described herein with reference toflowchart illustrations or block diagrams of methods, apparatuses(systems), and computer program products according to embodiments of thepresent disclosure. It will be understood that each block of theflowchart illustrations or block diagrams, and combinations of blocks inthe flowchart illustrations or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe block(s) of the flowchart illustrations or block diagrams.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other device to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the block(s) of the flowchartillustrations or block diagrams.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other device to cause aseries of operational steps to be performed on the computer, otherprogrammable apparatus, or other device to produce a computerimplemented process such that the instructions which execute on thecomputer, other programmable data processing apparatus, or other deviceprovide processes for implementing the functions/acts specified in theblock(s) of the flowchart illustrations or block diagrams.

The flowchart illustrations and block diagrams in the Figures illustratethe architecture, functionality, and operation of possibleimplementations of systems, methods, and computer program productsaccording to various embodiments of the present disclosure. In thisregard, each block in the flowchart illustrations or block diagrams mayrepresent a module, segment, or portion of code, which comprises one ormore executable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order or out of order, dependingupon the functionality involved. It will also be noted that each blockof the block diagrams or flowchart illustrations, and combinations ofblocks in the block diagrams or flowchart illustrations, can beimplemented by special purpose hardware-based systems that perform thespecified functions or acts, or combinations of special purpose hardwareand computer instructions.

While the foregoing is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A wearable headset comprising: an opticalarrangement configured to couple light from a display onto a first fieldof view; one or more visual sensors defining a second field of view; andone or more computer processors configured to: receive a signalindicating that the headset is in an unworn state; and while the headsetis in the unworn state, operate the one or more visual sensors to detectone or more user inputs within the second field of view.
 2. The wearableheadset of claim 1, wherein the one or more visual sensors comprise acamera that has a forward-oriented field of view when the headset is ina worn state.
 3. The wearable headset of claim 1, wherein the signalindicates whether the headset is physically coupled with a dockingdevice.
 4. The wearable headset of claim 3, wherein the one or morecomputer processors are further configured to: while the headset is in aworn state, output visual content to the display; and while the headsetis in the unworn state, output the visual content to an externaldisplay.
 5. The wearable headset of claim 4, wherein the externaldisplay is communicatively coupled with the one or more computerprocessors via the docking device.
 6. The wearable headset of claim 1,wherein detecting the one or more user inputs comprises: visuallydetecting one or more hand-held or body-worn controller devices withinthe second field of view.
 7. The wearable headset of claim 1, whereinthe wearable headset is operable in one or both of an augmented reality(AR) mode and a virtual reality (VR) mode.
 8. A system comprising: adocking device; and a wearable headset comprising: an opticalarrangement configured to couple light from a display onto a first fieldof view; one or more visual sensors defining a second field of view; andone or more computer processors configured to: receive a signalindicating that the headset is docked with the docking device; and whilethe headset is docked with the docking device, operate the one or morevisual sensors to detect one or more user inputs within the second fieldof view.
 9. The system of claim 8, wherein the one or more visualsensors comprise a camera that has a forward-oriented field of view whenthe headset is in a worn state.
 10. The system of claim 8, wherein theone or more computer processors are further configured to: while theheadset is in a worn state, output visual content to the display; andwhile the headset is docked, output the visual content to an externaldisplay.
 11. The system of claim 10, wherein the external display iscommunicatively coupled with the one or more computer processors via thedocking device.
 12. The system of claim 8, further comprising: one ormore hand-held or body-worn controller devices, wherein detecting theone or more user inputs comprises: visually detecting one or morehand-held or body-worn controller devices within the second field ofview.
 13. The system of claim 12, wherein the one or more controllerdevices are configured to dock with the docking device.
 14. The systemof claim 13, wherein the docking device is configured to: deliverelectrical power to one or both of the headset and the one or morecontroller devices, when docked with the docking device.
 15. A methodfor use with a wearable headset, the method comprising: receiving asignal indicating whether the headset is an unworn state; while theheadset is in a worn state: outputting visual content to a display ofthe headset, wherein the headset further comprises an opticalarrangement configured to couple light from the display onto a firstfield of view; and while the headset is in the unworn state: operatingone or more visual sensors of the wearable headset to detect one or moreuser inputs within a second field of view of the one or more visualsensors.
 16. The method of claim 15, wherein the one or more visualsensors comprise a camera that has a forward-oriented field of view whenthe headset is in the worn state.
 17. The method of claim 15, whereinthe signal indicates whether the headset is docked with a dockingdevice.
 18. The method of claim 17, further comprising: while theheadset is in the unworn state, outputting the visual content to anexternal display.
 19. The method of claim 18, wherein the visual contentis output to the external display via the docking device.
 20. The methodof claim 15, wherein detecting the one or more user inputs comprises:visually detecting one or more hand-held or body-worn controller deviceswithin the second field of view.